PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Models for Low-Rank Video Representation and Reconstruction

Rakib Hyder, M. Salman Asif

arXiv:1902.11132v1cs.CVcs.LGstat.ML

TL;DR

Video reconstruction from very few measurements requires compact representations that capture spatial and temporal redundancy. This paper uses a deep generator with latent-code optimization, optionally updating its weights, and regularizes neighboring codes with similarity and low-rank constraints. Experiments report reconstruction from compressive measurements and suggest latent interpolation can generate missing frames.

  • Problem

    Recovering videos from very few measurements is underdetermined, motivating compact representations that exploit video redundancy.

  • Method

    The method represents frames with generator outputs from latent codes, optimizes the codes or jointly optimizes codes and generator weights, and imposes similarity and low-rank constraints.

  • Results

    The experiments show video sequences can be reconstructed from compressive measurements using latent-code optimization or joint optimization of latent codes and network weights.

  • Takeaways & Limitations

    Low-rank latent embeddings can represent video sequences compactly and can potentially support generating new frames similar to an observed sequence.

  • Takeaways & Limitations

    The method’s success is attributed to good initialization and a hypothesized set of good local minima near the initial weights; proving this remains future work.

Abstract

from arXiv · show

Finding compact representation of videos is an essential component in almost every problem related to video processing or understanding. In this paper, we propose a generative model to learn compact latent codes that can efficiently represent and reconstruct a video sequence from its missing or under-sampled measurements. We use a generative network that is trained to map a compact code into an image. We first demonstrate that if a video sequence belongs to the range of the pretrained generative network, then we can recover it by estimating the underlying compact latent codes. Then we demonstrate that even if the video sequence does not belong to the range of a pretrained network, we can still recover the true video sequence by jointly updating the latent codes and the weights of the generative network. To avoid overfitting in our model, we regularize the recovery problem by imposing low-rank and similarity constraints on the latent codes of the neighboring frames in the video sequence. We use our methods to recover a variety of videos from compressive measurements at different compression rates. We also demonstrate that we can generate missing frames in a video sequence by interpolating the latent codes of the observed frames in the low-dimensional space.

1 Introduction

The paper uses deep generative models to represent and reconstruct videos from few measurements, combining latent-code optimization with low-rank and similarity constraints across frames. Jointly optimizing latent codes and generator weights expands recovery beyond the pretrained generator’s range and supports generating related frames.

  • 1 Introduction: The method represents each video frame with a low-dimensional latent code and reconstructs it from compressive measurements through a deep generator.The generator maps latent codes to images, while the measurement model records undersampled linear observations.
  • 1.1 Motivation and Related Work: The paper frames video recovery as a compressive-sensing problem because very few measurements leave the unknown frames underdetermined.Classical approaches use signal structure such as sparsity or low rank as prior information.
  • 1.2 Main Contributions: Jointly optimizing latent codes and network weights reconstructs sequences outside the initial generator’s range while similarity and low-rank constraints regularize the solution.The authors report that joint optimization converges to good solutions despite the network’s many parameters.
  • 1.2 Main Contributions: Direct latent-code optimization recovers a video only when the sequence lies within the range of the pretrained generator.Updating the generator weights enlarges the recoverable image range compared with optimizing latent codes alone.
  • 1.2 Main Contributions: Low-rank and similarity constraints exploit redundancy among consecutive frames, reducing the number of latent parameters needed for reconstruction from few measurements.The approach assumes neighboring frames have related representations, with variation localized in a lower-dimensional latent space.
  • 1.2 Main Contributions: A low-rank latent structure can also provide a low-dimensional manifold for generating new frames similar to the observed sequence.The paper connects this capability to interpolating or transforming latent representations across video frames.

2 Technical Approach

The approach represents video frames through latent codes generated into images, then extends latent-only recovery with joint generator-weight optimization and latent-space regularization. Similarity and low-rank constraints exploit temporal redundancy, allowing a sequence to be represented with fewer orthogonal codes.

  • 2 Technical Approach: Video recovery optimizes latent codes so a pretrained generator maps them to frames consistent with the measured observations.The formulation treats each frame as x_t = G_γ(z_t) and measurements as y_t = A_tG_γ(z_t).
  • 2.1 Latent Code Optimization: Latent-only optimization is inadequate when target images fall outside the pretrained generator’s limited range.The limitation is tied to the diversity of possible images relative to what one or a few generators can represent.
  • 2.2 Joint Optimization of Latent Codes and Generator: Jointly optimizing latent codes and generator weights overcomes this range limitation, including for compressive recovery of sequences outside the pretrained model’s range.The recovered frames are generated as x̂_t = G_γ̂(ẑ_t), with weights initialized from the pretrained generator.
  • 2.2 Joint Optimization of Latent Codes and Generator: Figure 3 shows latent-only optimization failing on out-of-distribution Moving MNIST and Color wheel sequences, whereas joint optimization generates both.The additional generator parameters provide substantially more degrees of freedom than changing latent codes alone.
  • 2.3.1 Similarity Constraints: Similarity regularization assumes adjacent frames have similar latent codes, using weighted differences between neighboring codes to exploit temporal continuity.The method fixes adjacent-frame similarity weights to one when neighboring frames are assumed close.
  • 2.3.2 Low Rank Constraint: The low-rank constraint restricts the latent-code matrix Z to rank r, implemented by retaining its top r singular vectors at each iteration.Each latent code is expressed as a linear combination of r orthogonal basis vectors, representing a T-frame sequence with r codes.
  • 2.3.2 Low Rank Constraint: Algorithm 1 alternates latent-code updates, rank projection, and network-weight updates until convergence or the maximum number of epochs.It takes measurements, measurement matrices, and a pretrained generator as inputs, and outputs latent codes and generator weights.

3 Experiments

The experiments evaluate the recovery method across measurement settings, datasets, and video types. They use pretrained DCGAN-style generators, synthetic and real sequences, and PSNR to quantify reconstruction quality.

  • Choice of generator: The generators follow DCGAN architecture without batch normalization, using strided convolutions, ReLU intermediate layers, and Tanh output.The latent code is projected and reshaped before the deconvolutional layers.
  • Initial generator training: Generator training jointly updates network parameters and latent codes with SGD and squared reconstruction loss.The experiments use minibatches of 256, with separate RGB and grayscale generators trained on resized CIFAR10 and MNIST images.
  • Measurement matrix: Experiments use identity measurements, Gaussian compressive measurements, and randomly missing pixels to test recovery across inverse-problem settings.Independent measurement matrices are generated per frame for compressive experiments.
  • Datasets: The evaluation spans MNIST digits, Moving MNIST, synthetic rotating color wheels, KTH walking video, and UCF101 archery video.Sequences range from 20 to 80 frames, with synthetic rotations and cropped real-video examples.
  • Performance metric: Recovery performance is measured using peak signal-to-noise ratio (PSNR), defined from image dynamic range and mean squared error.PSNR compares each reconstruction with its corresponding image.

4 Results

Across compact representation, compressive recovery, manifold linearization, interpolation, and complex-motion embedding, the experiments show that joint latent-code and weight optimization, combined with low-rank or similarity constraints, supports reconstruction and structured latent representations.

  • 4.1 Compact Video Representation: Joint optimization of latent codes and network weights provides a significant reconstruction-PSNR gain over latent-code optimization alone.The comparison is reported for compact video approximation and is attributed to the additional degrees of freedom in joint optimization.
  • 4.2 Optimization over zt with Constraints: Low-rank constraints represent Rotating MNIST and Person Walking with 12.5% and 20% of total-frame latent-code parameters, respectively, while improving reconstruction under very low measurements.The selected ranks are 4 for Rotating MNIST and 16 for Person Walking.
  • 4.2 Optimization over zt with Constraints: Similarity or low-rank constraints significantly improve reconstruction quality for compressive sensing, while low-rank constraints also outperform full-rank recovery with fewer latent codes.The experiments cover Gaussian measurements and inpainting settings summarized in Table 2.
  • 4.3 Joint Optimization over zt and γ with Constraints: For joint recovery from linear measurements, rank-4 latent constraints provide a small reconstruction-quality improvement over full-rank recovery.Figure 5 compares original frames, unconstrained joint recovery, and rank-4 latent-code recovery.
  • 4.4 Linearizing Motion Manifold via Joint Optimization and Low Rank Constraint: A rank-2 PCA constraint maps rotating motion to a line in latent space, with average PSNRs of 26.8 dB for Moving MNIST and 33.6 dB for Rotating MNIST.For multiple Rotating MNIST sequences, different sequences align to different lines while preserving within-sequence order.
  • 4.5 Interpolation in Latent Space to Generate Missing Frames: Linear interpolation between latent codes generates missing frames when the sequence maintains global sequential order in its linear latent representation.For three Rotating MNIST sequences, frames 11–15 were withheld and intermediate frames were generated between frames 1 and 20.
  • 4.6 Low-Dimensional Embedding of Complex Motion: A rank-3 constraint embeds a rotating bunny sequence in a 2D plane while preserving its sequential order in latent space.The embedding uses the mean and first two principal vectors in a 3D space.

5 Discussion and Future Work

The paper concludes that generative models with constrained latent codes can reconstruct videos from compressive measurements and organize motion in low-dimensional spaces, while identifying initialization theory as future work.

  • 5 Discussion and Future Work: The proposed generative model reconstructs videos from compressive measurements through latent-code optimization or joint optimization of latent codes and network weights.The experiments also use similarity and low-rank constraints to regularize recovery and improve reconstruction quality.
  • 5 Discussion and Future Work: Joint optimization performs well despite extremely few measurements relative to network parameters, but the paper attributes this success to good initialization and hypothesizes nearby good weights.A proof of good local minima around initialization is left for future work.

A.1 Image Inpainting on Additional Video Sequences

On KTH videos with 80% missing pixels, joint optimization substantially outperforms latent-code optimization, while rank-2 linearization achieves performance similar to full-rank recovery.

  • A.1 Image Inpainting on Additional Video Sequences: Joint optimization significantly outperforms latent-code optimization on KTH inpainting sequences spanning six video categories.The experiments use 80% missing pixels per frame.
  • A.1 Image Inpainting on Additional Video Sequences: Rank-2 linearization achieves performance similar to full-rank joint recovery on the KTH inpainting sequences.Figure 10 shows reconstructions with rank=2 and a linearization constraint using PCA.

A.2 Untrained Network vs Pretrained Network for Initialization

Pretrained network weights provide a more reliable initialization than random weights for video inpainting, improving convergence behavior under severe undersampling.

  • A.2 Untrained Network vs Pretrained Network for Initialization: Pretrained weights converge faster than random initialization for Rotating MNIST and Handclapping inpainting with 80% missing pixels.The comparison uses reconstruction-loss curves over iterations.
  • A.2 Untrained Network vs Pretrained Network for Initialization: Random initialization can show false convergence on Rotating MNIST before eventually converging.
  • A.2 Untrained Network vs Pretrained Network for Initialization: For datasets such as Moving MNIST, untrained weights may make convergence difficult, motivating pretrained weights as the initialization strategy.

A.3 Network Parameters

The experiments use separate deconvolutional generators for RGB and grayscale images, with different latent dimensions, channel widths, and parameter counts.

  • A.3 Network Parameters: RGB and grayscale images use separate generator networks with 4 × 4 deconvolutional filters.
  • A.3 Network Parameters: The RGB generator uses a 256-dimensional latent code, while the grayscale generator uses a 32-dimensional latent code.The codes are projected and reshaped into 512 × 4 × 4 and 256 × 4 × 4 tensors, respectively.
  • A.3 Network Parameters: The RGB and grayscale generators contain 4,852,736 and 819,712 parameters, respectively.These totals are reported in Table 4.
Loading 1902.11132v1…